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CHALLENGE:  AN  EXPERIMENTAL  COMPUTER  PROGRAM  FOR  RESEARCH  ON 

ORGANIZING  KNOWLEDGE  TO  IMPROVE  RETENTION  AND  APPLICATION 


Requirement : 

This  effort  is  a  portion  of  work  initiated  in  response  to 
guidance  from  the  1983  TRADOC  Commander's  Conference  which 
emphasized  the  need  to /"improve  officers'  capability  to  think, 
plan  and  decide  in  developing  an  operations  concept.  At  the 
request  of  the  US  Army  Armor  School,  ARI  conducted  an  analysis  of 
the  Armor  Officer  Basic  Course  (AOBC)  to  identify  actions  the 
School  could  initiate  to  accomplish  this  objective.  A  major 
conclusion  of  this  analysis  was  that  student  officers  were  having 
difficulty  in  absorbing  and  organizing  the  large  amount  of 
information  presented  for  application  to  tactical  operations. 

ARI's  recommendation,  indorsed  by  the  Armor  School,  led  to  the 
development  of  computer-tutors  to  train  student  officers  to  apply 
the  types  of  cognitive  strategies  needed  to  organize  tactical 
knowledge  for  application.  These  computer-tutors  are 
interactive  videodisk  programs  that  are  scheduled  to  undergo  trail 
implementation  at  the  Armor  School  during  the  first  9  months  of 
1988.  The  CHALLENGE  program,  described  in  this  report,  was 
developed  as  an  experimental  technique  for  evaluating  the 
effectiveness  of  the  interactive  videodisk  programs  in  improving 
organization  of  the  student  officers'  tactical  knowledge. 


Procedure : 

CHALLENGE  is  a  theoretically  based  program  developed  for  the 
purpose  of  capturing  and  depicting  organization  of  a  student's 
knowledge  of  a  selected  knowledge  domain.^  This  program  has  been 
subjected  to  a  cyclic  series  of  pilot  runs  and  revisions.  Purpose 
of  these  pilot  runs  was  to  obtain  critiques  on  screens, 
instructions,  responses  required  of  user,  and  user's  attitudes 
toward  the  way  they  had  to  interact  with  the  program.  CHALLENGE 
is  now  being  submitted  to  experimental  testing  in  an  instructional 
setting  with  college  students. 


Findings : 

The  CHALLENGE  program  requests  the  subject  to  respond  with 
keywords  denoting  major  concepts  and  their  supporting  concepts 
they  consider  important  to  understanding  the  central  topic 
provided  by  the  program.  While  this  initial  task  is  perceived  as 
similar  to  outlining,  this  perception  changes  to  that  of  a 
networking  task  when  the  subject  is  asked  to  show  how  the  major 
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concepts  identified  interrelate  in  his  or  her  understanding  of  the 
central  topic.  Pilot  runs  resulted  in  extensive  revisions  of  the 
program  and,  most  importantly,  expanded  options  for  returning  to 
add,  delete,  or  change  earlier  responses. 


Utilization  of  Findings: 

CHALLENGE  will  provide  a  research  tool  for  investigating  how 
content  and  organization  of  knowledge  upon  entry  into  training 
affects  learning,  is  altered  by  training,  and  how  the  resulting 
type  of  organization  of  the  student's  domain-specific  knowledge 
affects  retention  and  application  to  domain-relevant  situations. 
If  successful,  computer-tutors  of  this  type  will  provide  training 
developers  and  instructors  with  programs  needed  to  improve 
students'  acquisition,  retention,  and  application  of  knowledge  to 
domain-relevant  situations. 


INTRODUCTION 
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CHALLENGE:  An  Experimental  Computer  Program  for  Research  on 

Organizing  Knowledge  to  Improve  Retention  and  Application 


INTRODUCTION 

In  current  training  and  educational  settings  instructors  and 
reference  texts  are  used  to  present  a  large  amount  of  information 
to  the  students  for  initial  learning.  However,  once  the 
information  has  been  presented,  the  actual  learning  depends  on  the 
extent  and  kinds  of  cognitive  processing  activities  the  student 
performs  on  the  information.  Typically,  instructors  are  very 
limited  in  the  time  they  can  devote  to  tutoring  students  to  make 
interrelationships  among  concepts  that  are  needed  to  understand 
higher-level  concepts  or  their  application  outside  of  the 
classroom.  As  a  result,  much  of  this  new  information  either 
simply  falls  out  of  students'  short  term  memory  or,  if  memorized, 
is  stored  as  discrete  facts.  When  stored  in  this  way,  the 
knowledge  lacks  the  interrelationships  that  will  be  needed  for 
later  retrieval  and  application. 

Research  to  address  this  problem  focuses  on  the  experimental 
development  and  effectiveness  of  a  computer-tutor  that  will  cause 
the  student  to  generate  and  defend  interrelationships  among  facts, 
procedures  and  concepts  needed  to  understand  and  apply  a  given 
topical  domain.  In  contrast  to  conventional  CAI,  Adaptive  CAI, 
and  Intelligent  CAI,  the  experimental  cognitive-processing-tutor 
will  not  present  the  new  lesson  material  for  the  given  topical 
domain.  Instead,  these  tutors  will  cause  the  student  to  perform 
the  types  of  information  processing  required  on  information 
received  in  their  regular  instructional  setting,  and  on  their  own 
prior  knowledge,  to  develop  well  organized,  goal  directed, 
knowledge  of  the  domain. 

RESEARCH  BACKGROUND 

There  is  an  extensive  research  literature  on  identifying 
organization  of  students'  knowledge  in  specific  instructional 
domains  that  has  been  developing  since  the  early  1960 's.  Review 
of  this  literature  reveals  five  general  methods  used  to  elicit 
subjects'  knowledge  of  topical  domains.  These  five  are: 

1.  Interview  and  "think  aloud"  techniques  (e.g.,  Collins, 
1977;  West,  Fensham,  &  Garrardet,  1985).  Research  based  on 
techniques  of  this  type  will  be  addressed  in  a  subsequent  phase  of 
this  research  dealing  with  design  of  a  tutor  component. 

2.  Word  association  techniques  (Johnson,  1964,  1965,  1967, 
1969;  Johnson,  Curran,  &  Cox  1971;  Shavelson,  1972,  1973,  1974; 
Shavelson  &  Stanton,  1975;  Preece,  1976;  Thro,  1978;  Reitman  & 
Rueter,  1980;  McKeithen  &  Reitman,  1981;  Champagne,  Gunstone,  & 
Klopfer,  1985;  Gray,  Mutter,  Swartz,  &  Psotka,  1986;  Naveh- 
Benjamin,  McKeachie,  Lin,  &  Tucker,  1986;  and  Mutter,  Swartz,  & 
Psotka,  1 987 )  . 


3.  Concept  comparison  techniques  to  rate  or  otherwise  identify 
distance  among  stimulus  concepts,  {  Johnson,  1967,  1969;  Johnson, 
Cox,  &  Curran,  1970;  Johnson  et  al . ,  1971;  Shavelson,  1974;  Wainer 
&  Kaye,  1974;  Shavelson  &  Stanton,  1975;  Fenker,  1975;  Preece, 

1976;  Diekhoff,  1983;  Stanners,  Brown,  Price,  &  Holmes,  1983; 

Brown  &  Stanners,  1983;  Champagne  et  al.,  1985). 

4.  Concept  mapping  or  networking  with  identified  linkages 
(Holley,  Dansereau,  McDonald,  Garland,  &  Collins,  1979;  Champagne, 
Klopfer,  Desena,  &  Squires,  1981;  Fensham,  Garrard,  &  West,  1981; 
Novak  &  Gowin,  1984;  Novak,  1985;  Champagne  et  al . ,  1985; 
Dansereau,  et  al.,  1985). 

5.  Kelly  repertory  grids  (e.g.,  Shaw  &  Thomas,  1978;  Shaw  & 
Gaines,  1982) .  This  work  is  based  on  the  personal  construct 
psychology  of  George  Kelly  published  in  1955.  Shaw  &  Thomas 
(1978)  describe  two  "...BASIC  computer  programs  to  elicit  and 
analyse  grids  easily  and  clearly"  (p.139) .  This  technique  is 
listed  here  because  of  its  apparent  relevance  to  our  present 
research  problem.  However,  the  senior  author  only  recently 
uncovered  this  literature  and  an  attempt  to  review  it  in  this 
paper  would  be  premature. 

Of  the  five  types  of  methods  listed  above,  discussion  will 
focus  only  on  research  with  word  association,  concept  comparison, 
and  concept  mapping  techniques.  Since  word  association  and 
concept  comparison  techniques  share  the  same  methodological 
problems  to  be  discussed  here,  they  will  be  treated  as  one  group 
of  techniques. 


word  Association  and  concept  Comparison  Techniques. 

The  majority  of  the  research  studies  reviewed  have  utilized 
word  association  techniques  or  a  variety  of  concept  comparison 
techniques  to  elicit  subjects'  knowledge  of  the  topical  domain. 

To  accomplish  this,  a  set  of  concepts  is  selected  to  represent 
important  nodes  in  the  structure  of  the  topical  domain  (approaches 
used  to  select  concepts  will  be  discussed  below) .  Thus,  all 
subjects  receive  the  same  set  of  concepts  as  stimulus  words  for 
use  in  word  association  or  concept  comparison  techniques.  The 
resulting  representation  of  the  organization  of  their  knowledge  is 
based  on  group  data.  In  these  studies,  word  associations  or 
concept  comparison  data  obtained  from  the  "experts"  are  generally 
used  to  provide  a  reference  or  criterion  for  use  in  judging 
adequacy  of  the  structure  produced  by  the  group. 

We  will  focus  on  four  problems  that  are  common  to  the  use  of 
word  association  and  concept  comparison  techniques  in  the  studies 
reviewed.  These  are  (1)  limitations  inherent  in  word  association 
and  concept  comparison  techniques,  (2)  the  type  of  instructions 
usually  given  the  subjects,  (3)  approaches  used  in  selecting 
stimulus  concepts,  from  the  topical  domain,  and  (4)  the  usefulness 
of  research  findings  based  on  group  data. 


techniques .  The  major  limitation  lies  in  the  use  of  measures  of 
interrelationships  among  stimulus  concept  words  to  infer  the 
presence  of  a  particular  type  of  meaningful  relationship.  This 
has  been  of  particular  concern  among  researchers  in  science 
education  (Stewart,  1979;  Sutton,  1980).  Johnson  et  al.  (1971) 
described  the  word  association  test  as  a  nonveridical  assessment 
instrument  similar  in  type  to  the  MMPI  with  its  value  dependent  on 
diagnostic  normative  data.  Driver  and  Erickson  (1983)  assert  that 
word  association  and  concept  comparison  techniques  elicit 
propositional  knowledge  without  the  links  to  specific  events  or 
phenomena  needed  for  "knowledge-in-action"  (p.43).  Thro  (1978,  p. 
972)  rejected  the  "cognitive  structure"  label  Shavelson  (1972,  p. 
226)  applied  to  the  representation  of  students’  knowledge  obtained 
from  these  techniques  and  applied  the  more  pragmatic  label, 
"associative  structures."  Stewart  (1979)  argued  that  these 
techniques  are  useless  in  research  on  the  teaching  of  science 
since  they  are  unable  to  deal  with  propositional  relationships 
between  concepts.  Amplfying  on  this  point,  Fensham  et  al.  (1981) 
pointed  out  that  "...In  science  learning  the  links  between 
concepts  are  precise  propositional  statements  that  have  very 
definite  meaning"  (p.  121).  In  short,  the  presence  of  the 
"correct"  associative  cluster  does  not  necessarily  mean  the 
student  has  established  the  interrelationships  which  define  the 
concept  and,  futhermore,  the  establishment  of  the  desired  concept 
does  not  guarantee  its  linkage  to  application.  In  contrast, 
research  utilizing  interview  techniques  (type  1,  above)  and 
concept  mapping  with  explanation  of  relational  links  (type  4, 
above)  has  focused  attention  on  students'  frequent 
"misconceptions"  of  these  interrelations. 


Type  of  instructions  given  the  subject.  The  problem  of 
inferring  a  particular  type  of  meaningful  relationship  is  further 
complicated  by  the  type  of  instructions  given  to  the  subjects. 
Shavelson  (1974,  1975)  pointed  out  that  theoretical  views  on 
retrieval  and  decision  processes  have  strong  implications  for 
interpreting  students'  responses  obtained  from  word  association 
and  concept  comparison  techniques.  Shavelson  (1974)  stated: 

The  first  is  the  importance  of  the  context  established 
by  the  instructions  to  the  subject.  They  must  be  suf¬ 
ficiently  detailed  and  clear  to  allow  the  subject  to 
direct  his  search  of  LTM  and  to  establish  a  criterion 
against  which  he  can  test  alternative  responses.  (p .  237) 

However,  Shavelson,  as  well  as  other  researchers  in  this  area, 
were  content  with  establishing  context  by  instructing  the  subject, 
for  example,  to  "think  like  a  physicist"  when  the  topic  domain  was 
Newtonian  mechanics.  Champagne  et  al.  (1985)  pointed  out  that 
cognitive  structures  derived  from  data  collected  using  word 
association,  or  concept  comparison  techniques  represent: 


...  a  non-contextual  structural  organization  of  con¬ 
cepts.  Although  the  structures  obtained  by  these 
measures  are  in  one  sense  contextual  because  the 
students  are  well  aware  that  the  content  is  science 
or  physics,  they  are  non-contextual  in  the  sense  that 
the  concepts  are  not  being  applied  to  the  solution 
of  a  specific  task,  for  example,  the  comprehension  of 
text  or  the  solution  of  a  numerical  or  qualitative 
problem,  (pp .  165-166) 

In  other  words,  tl  subject  is  being  asked  to  think  about  the 
relevance  of  the  stimulus  words  to  a  given  topic  domain  (e.g., 
physics) ,  but  is  not  given  an  objective  or  purpose  the  knowledge 
to  be  retrieved  should  serve.  Thus,  it  seems  likely  that  for  any 
given  topical  domain,  the  specific  associative  structure  elicited 
from  the  student  will  vary  with  variations  in  retrieval  objectives 
given  in  the  instructions.  The  ambiguous  retrieval  objective  and 
the  quick  association  procedures  used  in  word  association  and 
concept  comparison  techniques  appear  to  purposively  minimize 
consciously  directed  thinking.  It  seems  reasonable  to  assume  that 
knowledge  structures  obtained  by  maximizing  consciously  directed 
thinking  are  more  likely  to  reflect  how  students  organize  their 
knowledge  for  application  than  the  structures  obtained  from  the 
techniques  reviewed  above. 

Appx,oa,,ch^s,.- used  in.  selecting  .stimulus,,  concep-ts  ..from.  the_topicai 
domain .  It  is  interesting  that  there  is  very  little  discussion  in 
this  literature  on  strategies  used  in  defining  a  topical  domain 
for  investigation  and  strategies  for  selecting  concepts  to  use 
from  that  domain.  In  fact,  instead  of  proceeding  in  a  top-down 
fashion  (identifying  the  structure  of  the  topical  domain  and  then 
selecting  concepts),  researchers  have  proceeded  in  an  inverse 
direction,  selecting  concepts  and  then  letting  these  concepts 
define  the  topical  domain.  The  usual  rationale  given  is  limited 
to  statements  indicating  that  concepts  were  selected  by 
instructors  or  experts  in  the  scientific  field  either  as  being 
important  in  understanding  that  topical  domain  or  as  being  a 
representative  sample  of  concepts  taught  in  that  topical  area. 

"Important"  may  mean  high  in  the  hierarchy  if  the  structure 
can  be  represented  in  this  fashion.  What  would  constitute  a 
"representative  sample"  of  concepts  from  a  well  organized  topical 
domain?  Apparently  "experts"  were  never  requested  to  describe  the 
nature  of  the  interrelationships  among  the  concepts  selected  in 
this  fashion.  Instead,  "experts"  usually  completed  the  same  word 
association  or  concept  comparison  task  administered  to  the 
students.  The  resulting  organization  of  the  concepts  by  the 
"experts"  was  then  used  as  the  criterion  in  evaluating  students' 
organization  of  the  concepts.  Thus,  while  instructors  or  other 
experts  may  perceive  patterns  of  interrelationships  among  the 
concepts,  it  is  not  clear  what  types  of  interrelationships  they 
recognized,  or  that  this  structure  was  even  being  taught  in  the 
course . 


Usefulness  of  research  findings  based  on  group  data.  Analytic 
methods  based  on  group  data  are  undoubtedly  a  more  efficient  way 
of  reducing  data  than  methods  for  dealing  with  each  individual 
separately.  Naveh-Ben jamin  et  al.  (1986),  in  the  context  of  word 
association  techniques,  and  Barsalou  (in  press),  in  the  context  of 
research  on  intra  and  inter-concept  similarity,  discuss  the  loss 
of  information  and  the  misinformation  one  obtains  by  using 
techniques  involving  averaging  across  subjects.  Unless  subjects' 
responses  are  highly  homogenous,  a  representation  of  the 
organization  of  the  group's  knowledge  seems  devoid  of  meaning. 
Thus,  within  the  context  of  the  present  research  on  development  of 
a  computer-tutor,  it  is  important  to  focus  on  changes  in 
associative  structure  within  the  individual  rather  than  the  group. 

Concept  Mapping. or  Networking  with  Identification  of  Linkages 


A  variety  of  techniques,  which  we  will  generically  refer  to  as 
concept  mapping,  were  developed  by  .Rowell  in  1974-1976  (as  cited 
in  Novak,  1985)  and  by  Champagne  et  al.  (1981)  as  assessment 
techniques.  Champagne  et  al.  (1985)  have  continued  to  use  their 
technique,  called  Concept  Structure  Analysis  Techniques  (ConSAT)  , 
for  evaluative  purposes.  The  ConSAT  is  conducted  in  the  context 
of  individual  interviews.  The  student  is  given  a  set  of  cards, 
each  containing  one  concept  word  previously  selected  as  important 
to  the  topical  domain.  The  student  is  asked  to  arrange  these 
cards  on  a  large  sheet  of  paper  in  a  way  that  "shows  how  you  think 
about  the  words".  While  or  after  completing  the  arrangement,  the 
student  is  asked  to  explain  why  the  words  are  arranged  as  they 
are.  As  the  student  identifies  relationships  between  words,  the 
interviewer  connects  the  words  with  a  line  and  labels  the  line 
with  the  relationship  given  by  the  student. 

Shortly  after  Rowell's  development  of  concept  mapping, 
researchers  at  Cornell  University  begin  using  it  not  only  in 
evaluation  but  as  a  direct  instructional  technique.  Unlike 
Champagne's  technique,  concept  mapping  described  by  Novak  and 
Gowin  (1984)  is  performed  as  a  part  of  regular  classroom 
activities.  Students  are  taught  how  to  construct  concept  maps  by 
arranging  the  initial  key  concepts  from  a  given  topic  in  a 
hierarchial  fashion  and  then  identifying  the  interrelationships 
linking  these  concepts.  They  are  then  instructed  to  expand  the 
map  by  generating  additional  concepts  and  the  relational  linkages 
the  student  considers  important  to  the  given  topic.  The  concepts 
and  linkages  generated  by  the  students  are  discussed  and  critiqued 
in  the  course  of  teaching  the  topic.  Novak  (1985)  and  Novak  and 
Gowin  (1984)  report  applications  of  concept  mapping  in  classes 
ranging  from  early  school  grade  level  into  college  classes. 
However,  Novak  reports  that  evidence  for  the  effectiveness  of  this 
strategy  is  still  largely  anecdotal.  Research  applications  are 
too  limited  in  both  duration  and  scope  of  the  students'  classes  to 
produce  important  changes  in  students’  knowledge  structures  and 
attitudes  toward  learning.  Scoring  criteria  with  nominal 
weightings  are  described  in  Novak  and  Gowin  (1984).  Novak  (1985) 
reports  that  these  scores  measure  something  that  is  substantially 


different  than  what  is  assessed  in  the  Scholastic  Aptitude  Test 
(SAT)  or  course  exam  grades.  Novak  attributes  this  to  the  failure 
of  the  SAT  and  course  exams  to  assess  meaningful  learning. 

Holley  et  al.  (1979)  and  Dansereau  et  al.  (1985)  have  conducted 
research  on  the  effectiveness  of  a  technique  similar  to  concept 
mapping,  but  which  they  label  "networking" .  Students  were  taught 
networking  as  a  strategy  for  improving  comprehension  and  retention 
of  text.  As  in  concept  mapping,  as  described  by  Novak,  students 
were  taught  to  convert  text  into  hierarchically  organized  node¬ 
link  diagrams.  An  important  difference  however,  is  that  in 
networking  the  student  is  taught  to  identify  and  use  specific  link 
types  and  types  of  text  structure.  Research  findings  have  been 
mixed.  In  the  Holley  et  al.  (1979)  study  the  networking  group  was 
superior  to  the  control  group  on  recall  of  main  ideas  from  the 
experimental  text.  However,  no  difference  was  found  in  the 
Dansereau  et  al .  (1985)  study. 

These  techniques  represent  teaching  strategies  for  inducing 
active  learning  of  relational  structures  while,  at  the  same  time, 
providing  a  graphic  reflection  of  the  student's  progress.  While 
Dansereau  and  Novak  might  use  different  labels,  they  perceive 
their  techniques  as  training  students  in  metacognitive  strategies 
that  will  improve  their  ability  to  learn  as  opposed  to  simply 
techniques  for  representing  organization  of  knowledge.  Novak 
advocates  continuous  use  of  concept  mapping  in  instructional 
settings  to  develop  students'  conceptual  meaning  as  opposed  to 
rote  learning  of  specific  facts.  However,  Novak  (1985)  also 
reports  resistance  to  introducing  concept  mapping  into  a 
curriculum  dominated  by  rote  learning.  Many  teachers  saw  it  as  a 
unwelcome  diversion  from  teaching  "basic  facts."  Development  of  a 
concept  map  was  hard  work  and  many  students  didn't  like  to  do  it. 
Nevertheless,  students  did  develop  concept  maps  and  reported  the 
experience  helpful  in  identifying  and  understanding  main  points. 

Concept  mapping,  as  described  by  Novak  and  Gowin  (1984)  and 
Novak  (1985) ,  is  the  technique  most  directly  relevant  to  the 
rationale  underlying  the  development  of  the  computer-tutor 
discussed  in  this  paper.  In  addition  to  the  use  of  concept 
mapping  with  students,  Novak  and  Gowan  recommend  that  instructors 
use  concept  mapping  to  identify  the  hierarchial  structure  and 
important  relationships  in  the  subject  matter  when  planning 
instruction.  This  represents  one  method  for  defining  a  domain  and 
identifying  subdomains,  along  with  their  important  concepts,  a 
method  that  appeared  to  be  missing  in  the  word  association  and 
concept  comparison  research  reviewed  above. 


RESEARCH  OBJECTIVES 


This  research  is  designed  to  sequentially  address  two  major 
objectives . 

Phase  1.  Objective:  A  computer-based  program  capable  of 
capturing  and  depicting  organization  of  a  student's  knowledge  of 
selected  topical  domains. 

Phase  2  Objective:  A  computer-tutor  that  embeds  the  program 
developed  in  Phase  1  with  explicit  tutor  components  that  train 
students  to  apply  appropriate  cognitive  strategies  in  the  context 
of  the  given  domain. 

This  computer-tutor  will  provide  a  research  tool  for 
investigating  how  content  and  organization  of  knowledge  upon  entry 
into  training  affects  learning,  is  altered  by  training,  and  how 
the  resulting  type  of  organization  of  the  student's  domain- 
specific  knowledge  affects  retention  and  application  to  domain¬ 
relevant  situations.  If  successful,  computer-tutors  of  this  type 
will  provide  training  developers  and  instructors  with  programs 
needed  to  improve  students'  acquisition,  retention,  and 
application  of  knowledge  to  domain-relevant  situations. 

PHASE  1:  DEVELOPMENT  OF  THE  CHALLENGE  PROGRAM 

Design  Objectives 

The  objective  in  designing  computer  programs  for  use  in  this 
research  was  adopted  from  Wittrock's  generative  model  of  learning. 
To  quote  Wittrock  (1986) : 

According  to  my  model  of  generative  learning  (Wittrock, 

1981)  comprehension  is,  essentially,  the  generation  of 
a  structural  or  conceptually  ordered  representation  of 
the  relations  among  the  parts  of  the  information  to  be 
learned,  and  between  this  information  or  these  ideas 
and  one's  knowledge  base  and  experience,  (p.  308) 

Wittrock  distinguishes  between  "...generation  of  a  structural  or 
conceptually  ordered  representation  of  the  relations  among  the 
parts..."  (comprehension  and  knowledge  acquisition)  and  generation 
of  relations  between  information  to  be  retained  and  arbitrary 
mnemonic  "hooks"  to  achieve  recall  that  will  mirror  the 
information  presented. 

Our  goal  in  developing  the  CHALLENGE  program  is  to  provide  a 
context  for  interaction  with  the  computer  that  will  induce 
students  to  examine  interrelationships  they  can  produce  among 
information  received  in  their  regular  instruction  and  also,  to 
interrelate  this  information  with  their  own  prior  knowledge  and 
experience.  In  this  sense,  we  want  CHALLENGE  to  provide  a  context 
for  activating  in  the  students  a  metacognitive  mode  of  thinking 


about  how  they  are  developing  and  organizing  their  knowledge  about 
a  given  topic.  In  addition,  the  program  must  provide  feedback  in 
the  form  of  a  printout  of  the  nodes  and  leaves  students  have 
produced,  the  cross-links  between  nodes,  and  data  needed  to 
construct  a  graphic  representation  of  how  they  organized  their 
knowledge  during  the  session. 


Programming  of  CHALLENGE  programs 

Programming  has  been  accomplished  by  the  second  author  using 
the  Apple  II  Pascal  1.1  (based  on  UCSD  Pascal  2.1)  language. 

These  programs  run  on  an  Apple  lie  with  two  disk  drives  and  a 
printer.  The  discussion  which  follows  will  describe  the  initial 
version  (CHALLENGE  I) ,  summarize  results  obtained  during  pilot 
runs  which  led  to  the  revised  version  (CHALLENGE  II),  and, 
finally,  will  describe  CHALLENGE  II  and  its  associated  analysis 
programs.  CHALLENGE  II  is  the  version  that  will  be  used  in 
research  applications  in  a  university  setting  during  the  1987  - 
1988  academic  year. 

CHALLENGE I 

Overview  of  CHALLENGE  I .  CHALLENGE  I,  the  initial  version, 
is  actually  an  umbrella  title  for  a  basic  program  that  went 
through  a  series  of  revisions  of  screen  designs,  wording  of 
instructions,  and  the  mechanics  of  moving  through  the  program. 

The  initial  screen  informed  the  subject  that  the  purpose  of  the 
program  was  to  assist  a  person  in  thinking  about  information 
received  in  class  and  how  it  can  be  organized  to  achieve  better 
understanding  (of  the  central  topic  provided) .  The  program  then 
presented  an  overview  of  what  the  subject  would  be  asked  to  do. 

It  then  proceeded  to  ask  the  subject  to  list  the  major  concepts, 
procedures  or  things  "you  think  you  need  to  know  to  understand 
(the  central  topic)".  The  subject  could  enter  up  to  30  major 
concepts.  However,  if  nine  or  more  were  entered  the  program  would 
ask  the  subject  to  review  the  list  and  identify  the  terms  "that 
will  provide  the  best  summary  of  what  you  need  to  know  ..."  and 
limit  it  to  not  more  than  eight  terms.  Following  this,  the 
subject  was  asked  to  rank  order  the  major  concepts  in  terms  of 
their  importance  to  understanding  the  central  topic.  Next,  the 
subject  was  presented  with  the  first  major  concept  listed  and 
asked  to  list  concepts,  procedures,  or  things  "you  think  you  need 
to  know  to  understand  (the  major  concept  identified)."  Again,  the 
program  was  designed  to  accept  up  to  8  terms  generated  to  support 
each  major  term.  The  subject  was  asked  to  rank  order  this  second 
level  of  terms  to  show  their  relative  importance  to  his  or  her 
understanding  of  the  major  concept  for  which  they  were  generated. 
This  procedure  of  generating  second  level  terms  for  each  major 
concept  and  then  rank  ordering  the  second  level  terms  was  repeated 
for  all  of  the  remaining  major  concepts.  The  final  part  of  the 
program  asked  the  subject  to  rate  the  importance  of  each  second 
level  term  to  each  of  the  major  concepts.  A  printout  was  then 
obtained  showing  what  the  subject  had  produced  in  each  step  of  the 
program . 
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Pilot  runs  on  CHALLENGE  I.  Five  colleagues,  all  research 
psychologists,  completed  one  session  on  this  initial  version  of 
the  program.  The  purpose  of  these  pilot  runs  was  to  obtain 
critiques  on  screens,  instructions,  rankings  and  ratings  required, 
and  user's  attitudes  toward  the  way  they  had  to  interact  with  the 
program.  Since  the  colleagues  were  volunteers  in  a  work  setting 
rather  than  students  in  an  instructional  setting,  we  had  to  select 
a  topical  domain  that  would  represent  a  commonly  shared  knowledge 
base.  The  central  topic,  "Evaluating  a  basic  research  proposal" 
was  selected  based  on  the  subjects’  prior  research  training  and 
subsequent  job  experience  in  this  work  setting.  Note  that 
"evaluating"  is  the  goal  the  subject  was  given  for  retrieving  his 
or  her  knowledge  about  a  topical  domain  labeled  "basic  research 
proposals".  Instructions  used  in  the  bulk  of  the  earlier  research 
literature  reviewed  above  did  not  give  the  subject  a  specific  goal 
for  retrieving  knowledge  related  to  the  stimulus  words  provided. 
For  example,  we  would  expect  that  the  major  concepts  and 
supporting  points  generated  would  be  different  if  the  goal  for 
this  topical  domain  was  "plan"  a  basic  research  proposal.  We  hope 
to  use  CHALLENGE  to  explore  differences  in  organization  of 
knowledge  on  retrieval  to  satisfy  different  goals  such  as 
"explain"  topical  domain  X  versus  "apply"  to  solve  a  problem  or 
make  a  decision. 


Time  to  complete  the  program  ranged  from  about  30  to  75 
minutes.  Each  of  the  five  subjects  generated  a  different  number 
of  major  concepts  ranging  from  3  to  7 .  These  subjects  generated 
an  average  number  of  supporting  points  ranging  between  4  to  5  per 
major  concept.  The  terms  used  for  the  major  concepts  and 
supporting  points  differed  widely  among  the  five  subjects  both  in 
terms  of  levels  of  abstraction  and  thoroughness  of  coverage.  The 
interrelationships  of  major  concepts  were  obtained  by  rating  the 
importance  of  each  supporting  point  to  each  of  the  major  concepts. 
These  differences  in  terms  used  and  the  patterns  of 
interrelatedness  result  in  five,  essentially  idiosyncratic, 
organizations . 
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V. 


Subjects  made  clear  one  major  problem  they  had  in  interacting 
with  this  version  of  CHALLENGE.  The  absence  of  options  to  go  back 
and  delete,  change,  or  add  both  major  concepts  or  second  level 
terms  imposed  a  serious  handicap  on  the  subjects.  As  a  result, 
they  were  forced  to  retrieve,  organize  and  record  their  thoughts 
in  a  serial,  step-by-step  fashion,  without  the  option  of  going 
back  and  revising  previous  entries  to  make  them  mesh  with  the 
conceptual  organization  as  they  were  developing  it.  Asking 
subjects  in  a  work  setting  to  generate  major  and  supporting  points 
important  to  their  understanding  of  a  topical  domain  is  very 
analogous  to  asking  for  a  "of f-the-top-of-the-head"  first  draft. 

In  addition  to  the  lack  of  editing  options,  subjects  had  not 
been  given  advance  notice  of  what  the  central  topic  would  be. 
Presession  priming  through  individual  or  group  discussions  of  the 
central  topic  would  probably  have  been  helpful.  It  appeared  that 
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subjects  generally  considered  what  they  produced  in  this  session 
as  representing  the  beginning  of  thinking  or  brainstorming  about 
the  central  topic  as  opposed  to  representing  an  organization  of 
their  knowledge  of  evaluating  a  basic  research  proposal.  Thus, 
generation  of  higher  and  lower  level  concepts  in  this  first 
session  appears  to  represent  more  a  preorganization  retrieval  of 
concepts  without  the  opportunity  to  review  and  edit  to  arrive  at  a 
satisfactory  "fit.”  As  a  result,  CHALLENGE  I  was  extensively 
revised  to  provide  the  printouts  and  editing  options  during  the 
session  in  CHALLENGE  II  (described  below) . 

However,  even  with  these  options,  it  is  expected  that  an 
individual  will  require  more  than  one  session  on  the  same  topic 
before  producing  an  organization  that  the  individual  will  deem 
adequate.  This  expectation  led  to  the  design  of  the  SUB-ANALYSIS 
confidence  rating  program  as  one  approach  to  obtaining  evidence  of 
the  subject's  level  of  satisfaction  with  the  concepts  generated 
and  their  organization  as  depicted  by  the  CHAL-ANALYSIS  program. 

Pilot  runs  also  indicated  the  need  for  deleting  the  procedures 
for  ranking  major  concepts  and  lower  level  concepts  in  order  of 
their  importance.  Since  this  ranking  task  intervened  between  the 
generation  of  successive  lists,  it  was  perceived  as  slowing  down 
and  disrupting  the  continunity  of  the  thinking  process. 

Several  approaches  to  analyzing  the  data  produced  by  the  pilot 
runs  were  considered.  A  major  concern,  and  one  that  has  been 
addressed  by  Fensham  et  al .  (1981),  is  providing  data  feedback 
that  is  easily  interpreted  by  the  instructor  and  the  student. 

These  considerations  led  to  the  development  of  the  CHAL-ANALYSIS 
program  that  will  be  described  below. 

Finally,  pilot  subjects'  critiques  resulted  in  numerous 
revisions  in  the  wording  of  program  instructions,  prompt  symbols, 
and  screen  displays. 


Overview  of  CHALLENGE  II.  CHALLENGE  II  runs  on  an  Apple  He 
with  two  disk  drives  and  a  printer.  Developed  for  research 
purposes,  the  program  is  initiated  by  the  researcher  for  each 
student  session.  The  researcher  uses  2  disks,  either  the 
CHALLENGE  program  (if  it  is  the  student's  initial  session  on  the 
given  topic)  or  the  Re-CHALLENGE  disk  (if  it  is  a  repeat  session 
on  the  topic),  plus  a  student  disk.  When  the  CHALLENGE  disk  is 
booted  up,  the  first  screen  will  present  a  menu  as  follows: 

1.  Challenge  program. 

2.  Chal-analysis  program 

3.  Changer  program,  changes  the  mother  node 

4.  Sub-analysis  program 

5.  Formatter  program 

6.  Exit  to  the  Pascal  system 

Enter  choice  on  the  keyboard 


If  the  student  disk  has  not  been  formatted,  the  researcher  will 
first  choose  #5  and  the  formatter  program  will  accomplish  the 
formatting.  If  the  mother  node  (the  central  topic)  has  not  been 
entered  on  the  CHALLENGE  disk,  or  if  it  is  to  be  changed  for  this 
session,  the  choice  of  #3  will  result  in  a  request  for  the 
researcher  to  type  in  the  Central  Topic.  The  analysis  programs, 

#2  and  #4, .will  be  discussed  later.  Number  6  is  included  for 
convenience  in  identifying  files  on  the  student  disk. 

Running  the  CHALLENGE  II  program.  Assuming  the  foregoing 
details  have  been  taken  care  of,  the  researcher  will  choose  #1  to 
initiate  the  first  session.  The  initial  screens  from  the 
CHALLENGE  disk  are  to  allow  the  researcher  to  log  in  the  roster 
number  assigned  to  this  student,  the  session  number  for  this 
student,  and  the  date.  The  program  is  then  ready  for  the  student 
to  take  over. 

Initial  instructions  on  the  CHALLENGE  disk  provide  an  overview 
of  what  the  program  will  request  the  student  to  do.  The  student 
will  be  given  the  option  of  bypassing  these  instructions  if  he  or 
she  is  familar  with  the  program  because  of  previous  sessions  on 
different  topics.  First,  the  student  is  asked  to  type  in  the 
major  concepts  perceived  to  be  important  to  the  student's 
understanding  of  the  central  topic  given  by  the  program.  Prompts 
at  the  bottom  of  the  screen  allow  the  student  to  edit  the  list  and 
to  indicate  completion.  Next,  the  student  is  presented  the  major 
concepts  entered,  one  by  one  in  a  succession  of  screens,  and  is 
asked  to  list  supporting  points  he  or  she  considers  most  important 
to  understanding  the  relationship  of  each  of  the  major  concepts  to 
the  central  topic.  While  creating  each  list,  the  student  is  given 
the  option  of  changing  or  deleting  terms. 


When  the  student  completes  listing  supporting  points  to  the 
last  major  concept  the  program  automatically  provides  a  printout. 
This  printout  contains  instructions  for  reviewing  and  editing  the 
lists  (Table  1)  and  the  supporting  points  listed  under  each  major 
concept  (Table  2) .  The  student  is  instructed  to  review  the  lists 
for  terms  that  the  computer  has  identified  as  duplications.  If 
duplications  are  present,  the  student  is  requested  to  edit  the 
lists  so  that  words  conveying  the  same  meaning  are  used  only  once. 
In  addition,  the  student  may  add,  change  or  delete  either  major 
terms  or  supporting  points  before  proceeding  to  the  final  stage  of 
the  program.  The  student  also  has  the  option  of  obtaining  a 
printout  after  any  change  made . 


Table  1 

Printout  of  Instructions  for  Editing  Keywords 


INSTRUCTIONS  FOR  EDITING  KEYWORDS 

This  printout  identifies  the  central  topic.  It  also  lists  in 
separate  columns  each  major  point  and  its  supporting  points  that 
you  provided. 

Please  review  the  keywords  you  have  used  and  your  organization  of 
major  points  and  supporting  points  to  make  sure  this  outline 
conveys  your  intended  meaning. 

Each  keyword  can  be  used  only  once.  The  printout  will  mark  words 
that  are  very  similar  in  spelling  (*)  or  are  identical  (+) .  If 
some  of  your  keywords  are  marked,  review  and  decide  where  to 
change  words  or  delete  them  so  that  words  conveying  the  same 
meaning  are  used  only  once. 

To  enter  an  editing  change,  find  the  list  number  that  is  at  the 
top  of  the  list  you  wish  to  change.  When  you  enter  that  number 
on  the  keyboard,  the  list  you  selected  from  the  printout  will 
appear  on  the  screen.  The  screen  will  look  like  it  did  when  you 
first  created  this  list  and  you  will  use  the  keyboard  commands, 
as  you  used  before,  to  change,  delete,  or  add  terms  to  the  list. 

After  you  have  entered  the  changes  for  a  list  you  can  choose  to 
edit  other  lists  or  to  obtain  a  new  printout.  You  may  edit  and 
re-edit  any  list  as  often  as  you  like.  When  you  have  finished 
editing  on  the  computer,  make  a  final  printout  of  your  work. 

If  you  have  any  questions,  please  ask  the  experimenter  for  help. 


Printout  of  Lists  Generated 
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Finally,  the  student  is  asked  to  show  how  the  ma^or  concepts 
identified  interrelate  in  his  or  her's  understanding  of  the 
central  topic  by  rating  the  importance  of  each  supporting  point 
to  each  major  concept.  Upon  completion  of  these  ratings,  the 
student  is  provided  with  a  printout  (Table  3)  containing 
instructions  for  reviewing  and  going  back  to  change  ratings  made 
in  any  of  the  lists.  The  second  part  of  this  printout  (Table  4) 
shows  all  of  the  supporting  points  generated  in  rank  order  by 
rating  score  under  each  major  concept.  When  the  student 
is  satisfied  with  the  rating  of  importance  of  each  supporting 
point  to  each  of  the  major  concepts,  a  revised  printout  is 
produced  and  the  SUB-ANALYSIS  program  is  initiated. 


Table  3 

Printout  of  Instructions  for  Editing  Ratings 


INSTRUCTIONS  FOR  EDITING  RATINGS 

This  printout  identifies  the  central  topic.  In  this  printout  all 
of  your  supporting  points  are  listed  in  order  of  importance  under 
each  of  the  major  points. 

Next  to  each  supporting  point  are  two  numbers.  The  first  number 
indicates  the  strength  of  the  relationship  between  the  supporting 
point  and  the  major  point  in  the  column  heading;  the  number 
varies  from  0  (not  important)  to  40  (most  important) .  The  second 
number  indicates  the  list  of  supporting  points  in  which  that 
rating  may  be  found. 

Please  study  the  ratings  of  importance  to  determine  if  they 
correspond  to  your  understanding  of  the  importance  of  each 
supporting  point  to  the  major  point  in  the  column  heading. 

Circle  any  ratings  that  you  would  like  to  change  and  indicate 
whether  the  rating  should  be  increased  or  decreased. 

When  you  have  finished  reviewing  the  ratings,  enter  these  changes 
on  the  computer.  To  change  a  rating,  find  the  list  number  that 
is  next  to  the  rating  you  wish  to  change.  When  you  enter  that 
number  on  the  keyboard,  the  list  with  the  ratings  that  you 
previously  entered  will  appear  on  the  screen.  Use  the  arrow 
keys,  the  Delete  key  and  the  number  keys  to  delete  the  old 
ratings  and  to  enter  the  new  ones  just  as  you  did  the  first  time. 

After  you  have  entered  the  changes  for  a  list  you  can  choose  to 
edit  other  lists  or  to  obtain  a  printout  of  the  revised  ratings. 
You  may  edit  and  re-edit  any  list  as  often  as  you  like.  When  you 
have  finished  editing  on  the  computer,  you  will  be  given  a 
printout  of  your  revised  ratings. 


If  you  have  any  questions,  please  ask  the  experimenter  for  help. 
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The  Re-CHALLENGE  disk.  Re-CHALLENGE  differs  from  CHALLENGE 
only  in  that  it  loads,  from  the  student  disk,  the  lists  of  major 
concepts  and  supporting  points  generated  in  a  previous  session  on 
the  same  central  topic.  The  program  starts  by  presenting  the 
student  with  a  printout  presenting  the  previously  generated 
lists.  The  program  asks  the  student  to  consider  whether  or  not 
his  or  her -understanding  of  the  central  topic  has  changed  since 
the  previous  session.  The  student  is  encouraged  to  review  the 
major  concepts  and  supporting  points  generated  in  the  last 
session  and  to  make  whatever  changes  desired.  The  program 
proceeds  from  this  point  just  as  it  did  in  the  CHALLENGE  program. 
Ratings  of  each  supporting  point  to  each  major  concept  made  in 
the  previous  session  are  not  presented  but  must  be  done  anew. 

The  SUB-ANALYSIS  program.  The  objective  of  the  SUB-ANALYSIS 
program  is  to  obtain,  (1)  ratings  of  students'  confidence  in 
their  understanding  of  the  central  topic  and,  (2)  ratings  of  the 
extent  to  which  the  relative  importance  of  major  points  and  their 
interrelationships  reflect  students'  understanding  of  the  central 
topic.  Since,  generally,  there  are  large  individual  differences 
in  the  way  people  distribute  ratings  along  a  rating  scale,  the 
SUB-ANALYSIS  program  is  initiated  by  either  the  experimenter  or 
the  student  setting  a  cutting  point  identifying  the  level  of 
rating  a  term  must  have  received  to  be  considered  an  "important” 
relationship . 

The  student  is  first  asked  to  rate,  on  a  0-100%  scale,  the 
level  of  confidence  in  his  or  her  understanding  of  the  central 
topic.  Following  this,  SUB-ANALYSIS  then  presents  the  major 
concepts  generated  by  the  student,  listed  in  order  of  importance, 
and  a  rating  scale.  Order  of  importance  is  based  on  the  average 
rating  of  importance  (final  stage  of  CHALLENGE)  of  all  supporting 
points  to  each  of  the  major  concepts.  The  student  is  asked  to 
rate  the  extent  to  which  this  order  of  importance  reflects  his  or 
her  understanding  of  the  central  topic. 

Finally,  the  student  is  presented  a  series  of  screens,  each 
containing,  in  matrix  form,  a  set  of  supporting  points  (rows)  and 
all  of  the  major  concepts  (columns) .  Cell  entries  are  asteriks 
to  indicate  supporting  points  rated  above  the  selected  cutting 
score.  Each  screen  contains  a  rating  scale.  The  student  is 
asked  to  rate  the  extent  to  which  the  pattern  of  asterikes 
reflect  his  or  her  understanding  of  each  of  the  major  concepts. 

To  see  the  effects  of  different  cutting  scores,  the  student  may 
be  recycled  through  the  SUB-ANALYSIS  program  and  a  printout 
(Table  5)  will  be  automatically  produced  each  time. 


1  8 


Table  5 


Printout  from  SUB-ANALYSIS  Program  -  - 

The  Central  Topic  la  Evaluating  a  basic  research  proposal. 
#5:01.02. TEXT[ 1 6 ]  Is  the  File  name. 

9  OCT  87 
01 

01  02 


Percent  confidence  in  the  Central  Topic  - - - >  101 

Agreement  with  the  Rankings  — - - - - - — - — - >  94 

Ratings  on  the  Major  Points: 

Cutting  point  set  to - — - >  32.9 

Agreement  with  theorectical  implio.  - >  90 

Agreement  with  type  of  study - — — — - - >  30 

Agreement  with  research  lit. - >  10 

Agreement  with  design  — - - >  66 

Agreement  with  resources  req. - - - — >  32 

Agreement  with  relev.  to  army  need - >  68 


The  C HAL- ANALYSIS  program.  The  interrelatedness  ratings  from 
the  final  step  in  the  CHALLENGE  (or  Re-CHALLENGE)  program  are 
analyzed  by  the  CHAL-ANALYSIS  program  to  provide  a  representation 
of  the  organization  of  the  student's  knowledge  during  that 
generative  session.  This  analysis,  discussed  below,  shows  the 
relative  importance  of  each  major  concept  to  the  central  topic.  In 
addition,  it  identifies  the  extent  to  which  supporting  points, 
originally  generated  under  one  major  concept,  are  cross-linked  or 
embedded  in  the  understanding  of  other  major  concepts. 

The  CHAL-ANALYSIS  program  assumes  that  the  relative  importance 
of  each  major  concept  to  the  central  topic  can  be  reflected  by 
computing  the  unweighted  mean  rating  of  all  sets  of  supporting 
points  on  a  given  major  concept.  Thus,  it  is  assumed  that  the 
greater  the  average  rating,  the  higher  is  that  major  concept  in 
the  hierarchy  of  the  topical  domain  as  generated  by  that 
particular  student.  These  means  are  shown  in  the  CHAL-ANALYSIS 
printout  (Table  6)  in  the  bottom  row  of  the  first  block  of  figures 
labeled  "height  statistics."  The  cutting  point  chosen,  shown  on 
the  line  below  the  mean  ratings  for  each  major  concept,  is  not 
invoked  in  calculating  the  height  statistics. 

The  second  block  of  figures  in  Table  6  shows  the  number  of 
supporting  points  which  received  an  importance  rating  above  the 
cutting  point  on  each  of  the  major  concepts.  It  is  assumed  that 
major  concepts  are  interrelated  if  at  least  one  supporting  point 
generated  for  concept  A  is  given  an  importance  rating  above  the 
cutting  point  on  one  or  more  of  the  other  major  concepts.  It  is 
also  assumed  that  the  greater  the  number  of  supporting  points 
shared  by  two  major  concepts,  the  stronger  their  interrelation¬ 
ship  . 

It  is  assumed  that  interrelatedness  can  also  be  depicted  as  bi¬ 
directional,  uni-directional,  or  no-relationship.  Bi-directional 
means  that  one  or  more  supporting  points  originally  generated  for 
concept  A  are  subsequently  rated  as  important  for  understanding 
concept  B,  and,  vice  versa,  supporting  points  originally  generated 
for  concept  B  are  subsequently  rated  as  important  for  understand¬ 
ing  concept  A.  Uni-directional  means  that  one  or  more  supporting 
points  generated  for  major  concept  A  are  rated  as  also  important 
to  understanding  concept  B;  however,  supporting  points  generated 
for  concept  ■  are  not  rated  as  important  for  understanding  concept 
A.  No-relationship  simply  means  that  no  supporting  points 
generated  for  concept  A  are  rated  as  important  to  understanding 
concept  B,  and,  furthermore,  none  of  the  supporting  points 
generated  for  concept  B  are  rated  as  important  to  understanding 
concept  A.  These  interrelationships  can  be  derived  from  the  last 
two  blocks  of  figures  in  Table  6.  However,  they  are  explicitly 
tabulated  in  Table  7,  the  second  page  of  printout  from  the  CHAL- 
ANALYSIS  program.  Finally,  the  third  page  of  printout  from  the 
CHAL-ANALYSIS  program  (Table  8)  provides  the  complete  matrix  of 
supporting  point  ratings  (rows)  for  each  major  point  (columns) . 
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•Table  7 


Page  2  of  Printout  from  CHAL-ANALYSIS  Program 

File  Name:  #5: 01. 02, text 
Name: 

Date  data  w as  collected:  9  OCT  87 
Roster  Number:  01 
Session  Number: 01  02 

Central  Topic:  "Evaluating  a  basic  research  proposal" 
The  cutting  point  was  set  to  32.9. 

Directionality  listings. 

Bi-directional  Uni-directional  No-relation 

1- 1  6-4  3-5 

1  -  2 

1  -  3 
1  -  4 
1  -  5 
1  -  6 

2- 2 

2-3 
2-4 

2- 5 
2-6 

3- 3 

3-4 

3- 6 

4- 4 

4- 5 

5- 5 

5- 6 

6- 6 
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Table  8 

VI 

Rage  3  of  Printout  from  CHAL-ANALYSIS 

Program 

;  » 

Pile  Name:  #5:01 .02. text 

*  m  ( 

.V, 

Nane: 

v;. 

Date  data  was  collected:  9  OCT  87 

Roster  Number:  01 

Session  Number: 01  02 

i 

r-i  ~ 

Central  Topic:  "Evaluating  a  basio  research  proposal" 

• 

Source  Data  for  Computations 

v’.’S 

1 

2 

3 

4 

5 

6 

V' 

1.  theorectical  impllc. 

replicate  p rev.  findings 

40 

1 

39 

38 

0 

33 

> 

extend  existing  theory 

40 

1 

39 

38 

0 

33 

& 

develop  new  constructs 

40 

1 

39 

38 

0 

33 

ft 

scope  of  Implications 

40 

1 

39 

38 

0 

37 

ft 

research  objectives 

40 

35 

34 

39 

38 

37 

ft 

2.  type  of  study 

36 

14 

V, 

) 

laboratory 

37 

40 

35 

0 

ethnographic 

37 

40 

32 

36 

28 

0 

field-controlled 

34 

40 

33 

39 

38 

37 

3.  research  lit. 

40 

36 

•.V, 

sources  of  lit  reviewed 

39 

37 

4 

5 

-mm  1 

relev.  of  review  to  objectives 

38 

0 

39 

40 

1 

20 

a'* 

> 

lit.  pro/con  to  objectives 

39 

0 

40 

38 

1 

35 

il.  design 

34 

40 

1 

38 

V 

relev.  depenent  meas.  to  obj. 

39 

0 

V 

relev.  indepent.  meas  to  obj 

39 

0 

34 

40 

1 

38 

quality  of  measures 

39 

33 

0 

40 

1 

3* 

>2 

tech,  of  quanitative  analyses 

39 

40 

34 

38 

36 

0 

%  i 

tech  of  qualitative  analyses 

38 

39 

34 

40 

36 

0 

> 

%■  s 

approp.  of  setting  to  obj. 

38 

37 

1 

40 

0 

39 

approp.  of  Ss  to  obj. 

39 

0 

1 

40 

2 

38 

data  coll,  procedure 

37 

38 

1 

40 

39 

0 

5.  resources  req. 

”.>1 

dollars 

38 

36 

0 

37 

40 

39 

subjects 

39 

1 

0 

37 

40 

38 

l 

-•  J 

time 

0 

36 

1 

38 

40 

39 

operational  disruption 

0 

35 

1 

38 

40 

39 

-2: 

6.  relev.  to  army  need 

API  work  program 

39 

0 

38 

1 

37 

40 

army  problems 

38 

37 

0 

1 

39 

40 

* 

political  climate 

38 

0 

1 

2 

39 

40 

-V-J 

The  height  statistics  (Table  6)  and  the  directionality  listings 
(Table  7)  can  be  used  to  construct  a  graphical  representation  of 
how  the  student's  knowledge  of  the  topical  domain  was  organized 
during  this  session.  Currently,  the  graphics  must  be  done  by 
hand.  Figure  1  presents  an  example  of  a  graphical  representation 
of  how  a  subject  organized  his  understanding  of  knowledge  needed 
to  evaluate  a  basic  research  proposal.  In  this  example 
"important"  is  defined  as  supporting  point  ratings  of  33  (40 
maximium)  or  greater.  The  vertical  dimension  is  based  on  the 
unweighted  column  means  shown  in  the  first  block  of  figures  in 
Table  6.  Horizontal  distance  in  Figure  1  is  arbitrary  and  has  no 
meaning.  Connecting  lines  are  bidirectional  if  no  arrow  head  is 
attached;  unidirectional  if  an  arrow  head  is  present. 

Note  that  not  all  possible  bidirectional  connections  are  drawn. 
The  reason  is  simply  to  avoid  a  "can  of  worms"  drawing.  The  rule 
is  that  as  long  as  the  subordinately  rated  (lesser  unweighted 
means)  major  concepts  have  bidirectional  relationships  with  the 
dominant  (the  largest  unweighted  mean)  major  concept,  there  is  no 
need  to  draw  lateral  connections  among  these  subordinate  major 
concepts.  However,  all  unidirectional  connections  must  be  drawn. 

The  bracketed  representation  of  interrelationships  is  provided 
to  aid  in  the  interpretation  of  the  graphic.  The  major  concepts, 
identified  by  their  number  in  the  brackets,  are  ordered  from  left 
(most  qualifying  supporting  points)  to  right  (fewest  qualifying 
supporting  points) .  When  two  or  more  embedded  major  concepts 
possess  the  same  number  of  qualifying  supporting  points  they  are 
shown  in  parentheses  with  an  equal  sign.  In  Figure  1,  note  in  the 
bracketed  representations  of  relationships  among  nodes  (Figure  1), 
that  major  concepts  #1,  #2  and  #6  embed  one  or  more  supporting 
points  (rating  score  of  33  or  greater)  from  each  of  the  other 
major  concepts.  The  relationship  between  #6  and  #4  is 
unidirectional.  The  supporting  points  originally  generated  as 
important  to  understanding  #6,  "Relevance  to  Army  need"  were  not 
rated  important  (rating  scores  less  than  33)  to  understanding  #4, 
"Design”.  However,  some  of  the  supporting  points  originally 
generated  as  important  to  understanding  "Design"  were  rated  as 
important  to  understanding  "Relevance  to  Army  need".  Also  note 
that  the  bracketed  representation  make  clear  that  there  is  no 
direct  interrelationship  between  major  concept  #3,  "Research 
Literature",  and  #5,  "Resources  Required".  This  is  not  made  clear 
in  the  graphical  representation  due  to  the  effort  to  avoid  the 
"can  of  worms". 

The  major  shortcoming  of  Figure  1  is  that  it  does  not  show  the 
supporting  points  which  establish  the  interrelationships  among  the 
major  concepts.  Currently,  the  complete  matrix  of  rating  scores 
shown  earlier  in  Table  8  is  the  only  concise  way  to  present  this 
information . 
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Mean 
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Major  Nodes 

Relationships 
Among  Nodes 

35 

1.  Theoretical  implications 

1-[4-(2=3=6)-5) 

32 

4.  Design 

4-[1-5-(2*3)] 

28 

6  Relevance  to  Army  need 

6-[(  1  =4)-5-(2=3)] 

24 

3.  Research  literature 

3-[  1  -4-2-6] 

22 

5.  Resources  required 

5-[(4=6)-(  1*2)] 

20 

2  Type  of  study 

2-[4-5-(  1*3=6)] 

Figure  1.  Interrelationships  based  on  supporting  point  ratings  equal 
to  or  greater  than  33  (out  of  possible  40). 
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Mean 

Ratings 

Major  Nodes 

Relationships 
Among  Nodes 

35 

1.  Theoretical  implications 

1  -{4-(6*3)-(5=2)J 

32 

4.  Design 

4-[1-5-(2=3)] 

28 

6.  Relevance  to  Army  need 

6-[(4=5)-1-2] 

24 

3.  Research  literature 

3-M-6J 

22 

5.  Resources  required 

5-[(4=6)-(  1  =2)] 

20 

2.  Type  of  study 

2-[4-5-(3=6)] 

Figure  2  Interrelationships  based  on  supporting  point  ratings  equal 
to  or  greater  than  36  (out  of  possible  40) 


Figure  2  illustrates  the  use  of  the  CHAL-ANALYSIS  program  to 
explore  the  way  the  subject  uses  the  rating  scale  to  define 
"importance".  In  this  figure  the  cutting  score  for  supporting 
point  ratings  was  raised  to  36,  or  90%  of  rating  points  possible. 
It  can  be  noted  from  Table  8  that  this  person  tended  to  use  the 
extreme  ends  of  the  rating  scale  with  virtually  no  ratings  in  the 
middle  of  the  scale.  As  a  result,  lowering  the  cutting  score  from 
33,  shown  in  Figure  1,  to  20  did  not  require  any  change  to  Figure 
1.  Raising  the  cutting  score,  as  shown  in  Figure  2,  resulted  in 
an  increase  in  unidirectional  relationships.  Since  this  produced 
an  unidirectional  relationship  between  major  node  1  and  2,  it  was 
necessary  to  draw  in  the  bidirectional  relationships  connecting 
node  2  with  nodes  4,  6,  and  5.  Whether  or  not  height  ratings  and 
the  directional  relationships  shown  have  significance  for  the 
retrieval  search  this  individual  would  employ  in  a  similar  context 
remains  to  be  seen. 

SUMMARY 

CHALLENGE  II  is  a  program  undergoing  experimental  development . 
It  will  be  tested  in  research  designed  to  evaluate  its  usefulness 
in  assisting  students  to  organize  their  knowledge  of  topical 
domains.  By  the  same  token,  this  research  will  test  its 
effectiveness  in  representing  how  the  students  have  organized 
their  knowledge  during  a  given  session. 

The  CHALLENGE  II  program  does  not  yet  include  a  tutor 
component.  However,  the  generative  thinking  this  program  requires 
of  the  student  is  expected  to  provide  a  context  for  activating  a 
metacognitive  mode  of  thinking  about  how  he  or  she  is  developing 
and  organizing  knowledge  of  a  topical  domain.  This  technique  is 
expected  to  produce  an  organization  of  knowledge  that  is  different 
than  the  associative  structures  obtained  from  word  association  and 
concept  comparison  techniques.  Presenting  students  with  a  central 
topic  and  an  objective  to  satisfy  in  generating  the  retrieval  and 
organization  of  their  knowledge  is  expected  to  maximize 
consciously  directed  thinking.  In  doing  this,  the  student  is 
identifying  where  linkages  exist.  Currently,  labeling  the  type  of 
relational  link  thought  to  exist  between  concepts  must  be 
accomplished  by  querying  the  student  following  completion  of  the 
session  on  the  computer.  It  is  anticipated  that  labeling  of 
relational  links  will  become  an  important  feature  of  the  phase  2 
computer-tutor  program. 
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